Skip to content

Enable push down optimization by default#3366

Merged
LantaoJin merged 1 commit intoopensearch-project:feature/calcite-enginefrom
qianheng-aws:feature/calcite-engine-pushdown-fix
Feb 28, 2025
Merged

Enable push down optimization by default#3366
LantaoJin merged 1 commit intoopensearch-project:feature/calcite-enginefrom
qianheng-aws:feature/calcite-engine-pushdown-fix

Conversation

@qianheng-aws
Copy link
Collaborator

@qianheng-aws qianheng-aws commented Feb 28, 2025 •
edited
Loading

Description

In the previous implementation, we push down project/filter into OpenSearchProjectIndexScanRule without any updating on its digest, which uniquely identifies a RelNode. This causes an equivalent issue with the nodes of CalciteOpenSearchIndexScan and makes the different RelSets in the optimizer mess up inaccurately.

This PR fixes this issue by overriding the explainTerms method to reflect PushDownContext in its final computed digest. And enable the push down by default in our settings, since all tests work fine the feature.

./gradlew :integ-test:integTest --tests '*Calcite*IT' succeed locally.

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@qianheng-aws
Enable push down optimization by default
89df3cb 
Signed-off-by: Heng Qian <qianheng@amazon.com>
@LantaoJin LantaoJin added the calcite calcite migration releated label Feb 28, 2025
@LantaoJin LantaoJin merged commit 37df1bf into opensearch-project:feature/calcite-engine Feb 28, 2025
6 of 13 checks passed
@LantaoJin
Copy link
Member

LantaoJin commented Feb 28, 2025 •
edited
Loading

CalcitePPLJoinIT. testComplexAntiJoin
CalcitePPLJoinIT. testComplexCrossJoin
CalcitePPLJoinIT. testComplexInnerJoin
CalcitePPLJoinIT. testComplexLeftJoin
CalcitePPLJoinIT. testComplexRightJoin
CalcitePPLJoinIT. testComplexSemiJoin
CalcitePPLJoinIT. testJoinTwoColumnsAndDisjointFilters
CalcitePPLJoinIT. testJoinWithCondition
CalcitePPLJoinIT. testJoinWithTwoJoinConditions
CalcitePPLJoinIT. testNonEquiJoin

These ITs failed after merged this PR.
Some IT failed with Caused by: OpenSearchStatusException[OpenSearch exception [type=parsing_exception, reason=[terms] unknown token [END_ARRAY] after [country]]]
And others IT failed with different output ordering (may not a problem if we accept the the result without ordering) @qianheng-aws

@qianheng-aws
Copy link
Collaborator Author

qianheng-aws commented Mar 3, 2025

CalcitePPLJoinIT. testComplexAntiJoin
CalcitePPLJoinIT. testComplexCrossJoin
CalcitePPLJoinIT. testComplexInnerJoin
CalcitePPLJoinIT. testComplexLeftJoin
CalcitePPLJoinIT. testComplexRightJoin
CalcitePPLJoinIT. testComplexSemiJoin
CalcitePPLJoinIT. testJoinTwoColumnsAndDisjointFilters
CalcitePPLJoinIT. testJoinWithCondition
CalcitePPLJoinIT. testJoinWithTwoJoinConditions
CalcitePPLJoinIT. testNonEquiJoin

These ITs failed after merged this PR. Some IT failed with Caused by: OpenSearchStatusException[OpenSearch exception [type=parsing_exception, reason=[terms] unknown token [END_ARRAY] after [country]]] And others IT failed with different output ordering (may not a problem if we accept the the result without ordering) @qianheng-aws

Seems PredicateAnalyzer has problem on support expression OR. Create a new issue to track this here #3368

@LantaoJin LantaoJin mentioned this pull request Mar 3, 2025
Merged
7 tasks
penghuo pushed a commit to penghuo/os-sql that referenced this pull request Mar 12, 2025
@qianheng-aws @penghuo
Enable push down optimization by default (opensearch-project#3366)
8a9b591 
Signed-off-by: Heng Qian <qianheng@amazon.com>
penghuo pushed a commit that referenced this pull request Jun 16, 2025
@qianheng-aws @xinyual
Enable push down optimization by default (#3366)
3b8130f 
Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LantaoJin LantaoJin LantaoJin approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

Assignees

No one assigned

Labels

calcite calcite migration releated

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@qianheng-aws @LantaoJin